Information Geometry of Propagation Algorithms and Approximate Inference

Consider the inference problem of undirected graphical models[8, 9]. When the graph is tree, the Belief Propagation (BP) algorithm (J.Pearl[11]) is an efficient algorithm and the exact inference is computed. However, when the graph is “loopy,” and the loops are big, the exact inference becomes intractable. Besides sampling methods, such as MCMC, tractable approximate inference gives us one practical solution, and the “loopy BP” algorithm is one of the most successful methods. Recently, it is pointed out that the idea of loopy BP have been used many fields, for example, Bethe approximation[3] in statistical physics, and the decoding algorithms of Low Density Parity Check (LDPC) codes [4] and turbo codes [2] in error correction codes. Although we observe the loopy BP works well in many applications, its theoretical aspects are not fully understood. Among some theoretical studies of loopy BP, we have studied it from information geometrical viewpoint[6, 7]. In this abstract, we first summerize the results of [6, 7] by defining the problem and showing the properties loopy BP based on information geometry[1]. We further show its relation to other propagation algorithms, including the convex concave computational procedure (CCCP)[14] and Adaptive TAP approximation[10]